Listening. Learning. Leading ? 



POLICY NOTES 

News from the ETS Policy Information Center 



Volume 18, Number 1 



Policy Evaluation & Research Center 



Winter 2010 



Addressing Achievement Gaps 

After the Bell Rings: Learning Outside of 
the Classroom and Its Relationship to 
Student Academic Achievement 

Education reform began with the schools, as advocates pushed 
for rigorous academic standards, enriched curricula and 
accurate tests. Improve the schools, reformers thought, and the 
achievement gaps separating low-income minority students from 
their more affluent White peers would vanish. But more than 
two decades later, those gaps remain, and to some reformers, 

one reason is clear: the schools, 
where children spend only a fraction 
of each day, can’t do it alone. 

“Schools are essential, but we've got 
to do something about the rest of 
the lives of children,” psychologist 
and education researcher Edmund 
W. Gordon told an audience of 
Edmund Gordon 2 00 assembled for ETS’s recent 

achievement gaps symposium on 
out-of-school learning. “Since learners spend so much of their 
time outside of the institution that we call school, we’ve got to 
give a lot more attention to manipulating and exploiting the 
educative functions of living itself." 

“Schools are essential, but we’ve got to do something about the rest of 
the lives of children.” — Edmund Gordon 




For the past three years, Gordon and a group of colleagues 
convened by ETS and Columbia University's Teachers College, and 
sponsored by the College Board®, have met to discuss the nature 
and impact of children’s out-of-school learning experiences. The 
work of the National Study Group on Supplementary Education, 
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After the Bell Rings: 

Learning Outside of 
the Classroom and its 
Relationship to Student 
Academic Achievement 

Each day, children spend more hours outside 
of school than in it. Yet education reformers 
have principally targeted the classroom, 
paying relatively little attention to what goes 
on during out-of-school hours. Now, however, 
reformers are beginning to realize that closing 
the stubborn achievement gaps separating 
low-income minority students from their more 
affluent White peers requires new attention 
to the many varieties of out-of-school 
learning — tutoring, summer school, even 
educational television. Efforts to expand and 
coordinate such supplementary education 
programs face many of the same obstacles 
that bedevil classroom reforms: budgets are 
tight, alignment with other educational efforts 
is imperfect, and proving that new initiatives 
increase achievement can be problematic. 
Still, with a new presidential administration 
apparently committed to rethinking education 
policy, it may be an auspicious moment 
for carrying education reform beyond 
the classroom. 

The promise of supplementary education was 
the focus of “After the Bell Rings: Learning 
Outside of the Classroom and Its Relationship 

(continued, on page 11) 



whose policy statement is forthcoming, 
informed ETS’s conference “After the Bell Rings: 
Learning Outside of the Classroom and Its 
Relationship to Student Academic Achievement,” 
held October 5-6 at the Fairmont Hotel in 
Washington, D.C. The conference, whose roster 
of two dozen speakers included researchers, 
advocates and educational administrators, was 
the 1 2th in ETS's six-year series of “Addressing 
Achievement Gaps” symposia. The conference 
was co-sponsored by the Charles Stewart 
Mott Foundation and co-conveners included 
A Broader, Bolder Approach to Education; 
the After-School Corporation; the Institute for 
Educational Leadership and its Coalition for 
Community Schools; the National Council of 
La Raza; and the National Urban League. 

Out-of-school learning has always played an 
important role in children’s lives, conference 
speakers agreed — although they jousted 
gently, with each other and with audience 
members, over whether such education is best 
identified as “supplementary,” “supplemental,” 
“complementary” or "comprehensive.” Whatever 
the label, and however long-standing the issues, 
increased attention to achievement gaps has 
turned a brighter spotlight on the way students 
spend their non-school hours: a University of 
Pittsburgh study concluded that 57 percent of 
achievement stems from non-school factors, said 
conference speaker Pascal D. (Pat) Forgione 
Jr., Executive Director of ETS’s Center on K- 12 
Assessment and Performance Management. 

The policy implications of such findings are 
manifold, conference speakers said. Once 
we acknowledge the impact of non-school 
influences on student achievement, some 
speakers argued, we must demand broadened 
access to such social supports as child care 
and health insurance — what Gordon called 
“a stabilization of the floor under existence.” 



Other speakers advocated lengthening the 
school day and year, to more closely match 
the time that most developed nations devote 
to education. And still other speakers stressed 
the importance of turning out-of-school hours 
into educationally productive time, especially 
for low-income children. Years ago, Gordon 
recalled, parents in Harlem told him they had 
done their part once they had gotten their 
children to school; meanwhile, parents in the 
affluent suburb where Gordon was raising his 
own family were filling their children’s days with 
tutoring and library visits. “The rest of our lives 
constitute important sources of education, and 
we’ve got to exploit them, improve them, utilize 
them, as much as we do schools themselves,” 
Gordon said. “And nobody needs to hear that 
message more than some of our economically 
disadvantaged, socially marginalized families.” 
Since low-income children often attend less 
effective schools, pointed out conference 
speaker Heather Weiss, Director of the Harvard 
Family Research Project, when they miss out on 
supplementary learning opportunities as well, 
“the debt compounds.” 

In a draft executive summary of its policy 
statement, circulated at the conference, Gordon's 
National Study Group on Supplementary 
Education calls for coordinated government 
and private-sector efforts to integrate in-school 
and out-of-school learning opportunities 
for all children. “Because of moral and legal 
imperatives, and in order to secure our economic 
future, it is now time to update and upgrade the 
ways that we prepare our children for life in the 
21st century,” the draft says. “In the hours that 
young people spend outside schools, we need to 
do better.” 

The draft lays out an ambitious agenda, calling 
for a reduction in school funding disparities 
among states; the development of comprehensive 
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out-of-school programs and curricula; an 
accountability system that broadens the 
definition of student achievement to include not 
only reading and math, but also such qualities 
as critical thinking and creativity; the funding 
and dissemination of research on effective out- 
of-school programs; and intensive efforts to 
increase parental and community involvement 
in children's development. As a first step, the 
draft proposes the creation of government- 
funded community resource centers that would 
help families and communities support child 
development and would identify and publicize 
local resources for supplementary education. 
“Just as health does not begin and end in 
hospitals, education does not begin and end 
in schools,” the draft says. “The nation must 
begin seeing public education as we see public 
health, as a comprehensive approach to social 
well-being.” 

Beyond One Size Fits All 

Across the country, conference speakers said, 
educators and advocates have developed a 
wide range of out-of-school programs, some as 
venerable as the 1 960s War on Poverty, some 
as newfangled as the most up-to-date website. 
These programs vary along many dimensions. 
They take place after school, in the summer, 
or whenever a child turns on a television 
or a computer. They are run by schools, in 
community centers or through universities. They 
serve different age groups, for different lengths 
of time. They incorporate academic preparation, 
college access, civic engagement, even just plain 
fun. Just as engineers build redundancy into 
mechanical systems — ensuring that, if one 
component fails, others will provide backup — 
so, too, supplementary programs that pick up 
where schools leave off can create a beneficial 
redundancy, offering many ways for children 



to acquire the skills, habits and knowledge they 
need for success, conference speakers said. 

“The array of educational opportunities, of 
development opportunities, is what’s important,” 
said Hal Smith, the National Urban Leagues 
Vice President for education. “There probably is 
not one model that is going to work across the 
age span, or even within a certain age band, to 
serve all of what young people need.” 

“The array of educational opportunities ... is what’s 
important. There probably is not one model that is 
going to work across the age span, or even within a 
certain age band, to serve all of what young people 
need.” — Hal Smith 



Some supplementary education programs take 
direct aim at test scores, speakers noted. Under 
the No Child Left Behind (NCLB) Act, schools 
that fail to meet achievement targets must 
offer their students tutoring, often supplied 
by commercial vendors under contract to 
school districts. Traditionally, summer school 
performed a similar catch-up function for 
struggling, often reluctant students. But now, 
inspired by data showing that all students — 
and especially low-income students — lose 
skills and knowledge over the summer, school 
districts and nonprofit organizations from 
Boston to Wyoming are striving to move 
beyond that “remedial, sort of punitive model,” 
said conference speaker Jeffrey Smink, Vice 
President of policy for the National Summer 
Learning Association. The most successful 
summer programs mix academic enrichment, 
field trips, mentoring and community 
engagement, he said, to produce something 
that looks and feels different from school. 

With a Baltimore study attributing two-thirds 
of the ninth-grade test-score gap in reading to 
summertime effects, “summer represents an 
untapped resource for closing the achievement 
gap,” Smink said. 





Why Summer Learning? 



• Summer Slide 

-39 empirical studies find evidence of 
summer learning loss 

- Most students lose two months of 
math skills 

- Low-income students lose two months 
in reading achievement while middle- 
class peers make gains 

• Achievement Gap 

- Two-thirds of the ninth-grade reading 
achievement gap can be explained by 
unequal access to summer learning 
opportunities 

- Summer learning loss has negative 
impact on high school graduation and 
college attendance rates 

Source: National Summer Learning Association 

Supplementary education programs also seek to 
keep low- income and minority students on track 
to high school graduation and beyond, speakers 
said. The National Urban Leagues Project Ready, 
run at 16 affiliates around the country, stresses 
college preparation and access, Smith said. The 
federally funded TRIO programs, including 
1960s-vintage Upward Bound and Talent Search, 
offer a mix of summertime, weekend and after- 
school services — including tutoring, counseling 
and mentoring — to disadvantaged students, 
especially those whose parents did not attend 
college. Studies of some TRIO programs have 
shown that participating students are more likely 
to continue their education after high school, 
said conference speaker Ngondi A. Kamatuka, 
the Director of educational opportunity 
programs at the University of Kansas. 



Data also testily to the effectiveness of the 
community school model of supplementary 
education, under which the school becomes a 
center for academic, health and social services, 
said conference speaker Martin J. Blank, 
President of the Institute for Educational 
Leadership and director of its Coalition for 
Community Schools. A community school 
builds partnerships with other organizations, 
and it stays open past the final bell of the school 
day, hosting after-school programs, internships 
and neighborhood gatherings. Community 
schools build on the insight that low-income 
students often lack social capital, the network 
of relationships that support learning, Blank 
said. “What community schools do is recreate 
those connections,” he said, “because they have 
universities and community-based organizations 
and hospitals and families all working together.” 

“Summer represents an untapped resource for 
closing the achievement gap.” — Jeffrey Smink 

Such broadened forms of community 
cooperation are integral to the educational 
efforts of the National Council of La Raza, which 
has organized 85 of its local affiliates into an 
expanded learning network, said conference 
speaker Delia Pompa, La Raza's Vice President 
for education. At Alta Vista High School in 
Kansas City, Missouri, a La Raza after-school 
and summertime enrichment program exposes 
students to college and career choices, arranges 
internships, and offers courses on such “soft" 
job skills as showing up on time. The high 
school has a Hispanic graduation rate three 
times the national average, Pompa said. Other 
La Raza affiliates take different approaches to 
integrating in-school and out-of-school learning, 
she said: a Chicago community center keeps 
a parent liaison on the staff of its after-school 
program to help guide immigrant families 





through their contacts with school officials; 
and a Houston charter school shares a building 
with a community center, where students can 
take photography classes and join the champion 
drum line. 



What Happens in a 
Community School? 


After-school enrichment 


Early child 


and extended learning 


development 


Family 


Family and 


support 


community engagement 


Adult education 


Community-based 


and workforce classes 


learning 


Health, mental health and 


Violence 


prevention services 


prevention 



Source: Coalition for Community Schools 



Not all supplementary education comes 
in institutionalized packages, pointed out 
conference speaker Susan Zelman, Senior 
Vice President of the Corporation for Public 
Broadcasting (CPB): public television is an 
important source of supplementary education, 
especially for low-income children, who watch 
more TV than do their middle-class peers. For 
14 years, federal Ready to Learn grants have 
funded CPB efforts to improve the reading skills 
of young children in high-poverty broadcast 
markets, Zelman said. The initiative includes 
online courses for parents and caregivers, kid- 
friendly websites featuring educational games 
and activities, a curriculum based on the popular 
shows “Sesame Street” and “Between the Lions,” 
and outreach efforts to day care centers and 
after-school programs. Parents can even sign up 
to receive cell-phone messages in which cuddly 
Elmo from “Sesame Street” encourages adults to 
help children turn everyday objects into props in 
an alphabet game. 



Over the past four years, research into the 
effectiveness of the Ready to Learn efforts has 
shown that several public television programs 
help improve young childrens vocabulary, 
letter recognition and enthusiasm for reading, 
said conference speaker Barbara E. Lovitts, 
CPB’s Director of research and evaluation. And 
research into the literacy initiative's “social 
marketing” components — efforts to involve 
parents and caregivers in their children’s out- 
of-school learning — has found that some 
approaches work better than others. “Messages 
such as 'Read to your child for 1 5 minutes a 
day’ and ‘Help prepare your child for school’ are 
not messages that resonate with low-income 
parents,” Lovitts said. “Low-income parents 
do not see themselves as their children’s first 
teachers.” The new, improved messages, she 
said, stress that parents can help children learn 
around the clock (“Any time is learning time”) 
and that such help is critical for a future 
beyond school (“Preparing your child for 
success in life”). 

“Messages such as ‘Read to your child for 15 minutes 
a day’ and ‘Help prepare your child for school’ are not 
messages that resonate with low-income parents ... 
Low-income parents do not see themselves as their 
children’s first teachers.” — Barbara Lovitts 



But as educators and advocates try to squeeze 
education into every moment of the day, they 
must not lose sight of what may be squeezed 
out, warns the Michigan public television 
documentary “Where Do the Children Play?”, 
an excerpt from which was shown at the 
conference. Children once spent hours in 
unstructured play, often outdoors, building 
imaginary worlds, discovering nature and 
learning how to negotiate, compromise and 
follow rules. But especially in the affluent 
suburbs, children now spend most of their out- 
of-school time tethered to electronic media or 






participating in extracurricular activities planned 
and organized by adults. These changes have 
undermined children's health and impoverished 
their emotional lives, said conference speaker 
Gil Leaf, whose wife, University of Michigan 
lecturer and author Elizabeth Goodenough, 
wrote the book on which the Emmy 
Award-winning PBS documentary is based. 
“Imagination and creativity is going to be the 
competitive edge for America,” Leaf said, “and 
what we're doing instead is regimenting kids.” 

Deploying supplementary education to close 
achievement gaps could potentially reduce the 
time available for unstructured play even further, 
noted conference speaker Ronald Ferguson, a 
Harvard researcher. But psychologist Gordon 
insisted that this need not be so. “In the fullest 
conception of what education ought to be about, 
it would include children's play,” Gordon said. “I 
see us including what is already there, but trying 
to focus it more effectively on the academic and 
the personal development of kids.” 

Better Than Leftovers 

Although supplementary education programs 
take different forms, similar policy implications 
emerge across the spectrum, speakers said: 
successful programs develop in response to local 
needs, rely on strong community partnerships, 
and insist on high quality. “The further away that 
the intervention is initiated, the less likely that 
it will be well implemented,” argued Harvard 
researcher Weiss. And spending more time on 
education without improving the quality of 
instruction will not yield learning dividends, 
conference speakers warned. “If all we do with 
that time is reproduce what we do not do very 
well now, then all we're going to get is more 
of the same,” said Carlos Rodriguez, Principal 
Research Scientist at the American Institutes 
for Research. 



Public television's high production values 
show what supplementary education can be 
at its best, said conference speaker Dennie 
Wolf, of consultants Wolf Brown. Zelman set 
the conference audience’s feet tapping when 
she screened an excerpt from “The Electric 
Company,” a hip-hop-inflected salute to “silent 
E,” featuring a multiracial cast of young 
people dancing through the halls of an inner- 
city school. “What goes on in after-school or 
supplementary education doesn’t need to be 
composed of the leftover graham crackers from 
earlier in the school day,” Wolf said. “It needs 
to be as exciting as multimedia.” Indeed, some 
traditional schools could take pointers from 
supplementary education programs, speakers 
said. For the Baltimore students who learn math 
through a community gardening project that 
culminates in a farmers' market sale, “the fun is 
more obvious,” said Ann E. Chafin, Maryland’s 
Assistant State Superintendent of education. “I 
just wish it could be more obvious during the 
regular school day.” 

“What goes on in after-school or supplementary 
education doesn’t need to be composed of the 
leftover graham crackers from earlier in the school 
day. It needs to be as exciting as multimedia.” 

— Dennie Wolf 



But the imperative to produce high quality can 
sometimes conflict with the imperative to act 
locally, conference speakers acknowledged. In 
theory, programs developed in response to local 
needs are more likely to be effective; in practice, 
however, local programs are often poorly 
implemented. Urban districts seldom understand 
how to set up sophisticated mechanisms for 
keeping program implementation on track and 
correcting problems early, said ETS’s Forgione, 
a former urban superintendent. “Just because 
you’re doing a program doesn’t mean that 
you’re making a difference,” warned keynote 
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speaker Angela Glover Blackwell, the CEO of 
consulting firm PolicyLink. "Just because you 
care doesn’t mean that you solved the problem.” 
The solution to the conundrum, Blackwell said, 
is to build local capacity: the people to invest in, 
she said, are those “who, if they could produce 
the high quality, would have the most impact, 
because they’re closest to the problem, because 
they’re trusted in the neighborhoods, because 
these are their children.” 

Budget Blues 

Supplementary education initiatives face 
many of the same challenges that bedevil 
other education reforms, conference speakers 
said. Money problems top the list: although 
supplementary education programs, including 
summer school, were among the permitted 
uses for the billions allocated under the federal 
stimulus package, the national recession 
has battered budgets everywhere. And while 
supplementary programs vary widely in cost, 
the most intensive do not come cheap: the pre- 
college Upward Bound program costs more 
than $4,000 per student per year, said Kansas 
university official Kamatuka, compared with 
only $400 per student for the far more limited 
Talent Search program. NCLB-required tutoring 
for students in struggling Milwaukee schools 
costs up to $2,000 per child, said conference 
speaker Robert H. Meyer, a research professor 
at the University of Wisconsin-Madison, who has 
evaluated Milwaukee’s tutoring initiative. And 
four studies conducted over the past year found 
that supplementary programs for elementary- or 
middle-school students cost an average of $2,600 
per child during the school year and $ 1 ,000 per 
child during the summer, for an average hourly 
cost of $7 per student, said conference speaker 
Sharon Deich, Vice President of the education 
policy consulting firm Cross and Joftus. High 



school programs were more expensive — $ 1 0 per 
student per hour, she said. 

Supplementary education programs draw on 
a range of public funding sources, Deich said, 
including education and social service budgets at 
the federal, state and local levels, many of which 
have been hit hard by the national recession. 

The pending reauthorization of the Elementary 
and Secondary Education Act may offer the 
possibility of new funding for supplementary 
programs, she said; less certain is the impact 
of the $4.35 billion in federal “Race to the 
Top” grants, available to states. But private 
money — whether from foundations or from 
businesses, which spent $136.6 million on local 
after-school programs in 2005 — has been “the 
driver of innovation around out-of-school-time 
programming,” Deich said. 



Challenge — Making Expanded 
Learning a Regular Part 
of Education Reforms 



School and city budgets are under siege 



Need grassroots as well as grasstops 
support to expand and sustain this work 



Need better research to show the value 
of expanded learning — that it is a 
cost-effective way to enhance education 



Need support for partnerships — funding for 
intermediaries, and other infrastructure 



Need a permanent home for 
expanded learning programs 



Source: Sharon Deich 





Alignment Issues 

However they are funded, supplementary 
education programs often face another problem: 
the lack of alignment with regular educational 
offerings, which can dilute the educational 
power of supplemental programs, speakers said. 
Sometimes, this lack of alignment is deliberate: 
the tutoring required under NCLB, often 
provided by vendors promoting a particular 
instructional approach, is intended to contrast 
with the regular instruction that has failed 
to raise student test scores, said Wisconsin 
researcher Meyer. The funding provided for 
alignment efforts is “zero,” Meyer said. “The 
program is conspicuously designed not to be 
aligned.” To some extent, such variability may 
be inevitable, said conference speaker Steven 
M. Ross, a senior research scientist at Johns 
Hopkins University, who has evaluated tutoring 
programs in 14 states: tutoring a struggling sixth 
grader may require teaching first-grade material, 
not the sixth-grade curriculum the student 
can't yet master. But lack of alignment with the 
increasingly responsive classroom evaluation 
systems that some schools have instituted gives 
vendors no way of assessing their impact on the 
classroom, Meyer said. “If you cut off the vendor 
from that source of information, you're making 
it a lot harder for them to respond and to know 
whether they're succeeding,” he said. “It’s a lot 
to ask a vendor, to build a completely separate 
replication of that information system. It doesn’t 
seem like it’s too efficient.” But to the extent 
that alignment is possible and desirable, it must 
be pursued carefully, said Harvard researcher 
Weiss, to avoid turning supplementary programs 
into clones of their school-day counterparts. “It’s 
going to be critically important how we think 
about aligning them with schools, so we don’t 
lose what’s so important about out-of-school 
time now,” she said. 



Data and Common Sense 

In the push to expand and coordinate 
supplementary learning opportunities, data 
showing which programs work will play a crucial 
role, conference speakers agreed. And across the 
country, despite the persistence of failure, some 
high-poverty schools are succeeding in educating 
low-income students, noted conference keynote 
speaker Pedro Antonio Noguera, a professor 
of education at New York University. “In public 
education, common sense is almost absent 
altogether,” Noguera said. “Why don’t we start by 
doing more of what works?” 

But for some 
supplementary 
programs, evidence 
of effectiveness 
is elusive or 
highly contested, 
conference 
speakers said. 

An independent 
evaluation of 
Upward Bound, 
which reported 
that the college- 
readiness program had anemic effects, was 
mired in controversy from the start, said 
conference speaker Margaret Cahalan of the 
U.S. Department of Education (ED). In 2008, 

16 years after the study began, Cahalan made 
public her doubts about technical aspects of 
its methodology, including the composition 
of the control group and the validity of the 
statistical weighting. In her re-analysis of the 
data, designed to eliminate what she saw as 
bias, Cahalan found that, eight years after 
high school, 21 percent of Upward Bound 
participants, but only 14 percent of the control 
group, had earned bachelor’s degrees; she 




Pedro Noguera 
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estimated that the government got back $5 
in higher tax payments for every $ 1 spent on 
the program. The argument was more than a 
tedious bureaucratic struggle among number- 
crunchers, Cahalan said, because the original 
findings — that the program had some effect for 
at-risk students but little effect overall — were 
used to justify ultimately unsuccessful efforts 
to eliminate Upward Bound’s funding. “It’s one 
of the few evaluations that did not get put on 
the shelf but was actually used to influence ED 
policy,” Cahalan said. 

Complications also plague evaluations of NCLB- 
required after-school tutoring, said Wisconsin 
researcher Meyer and Johns Hopkins researcher 
Ross. Although advocates hoped the program — 
formally known as Supplemental Educational 
Services, or SES — would yield big increases 
in test scores, Meyer said his evaluation of 
tutoring provided by commercial vendors in 
Milwaukee produced “the disappointing finding 
that we haven’t found gangbuster effects.” Ross' 
evaluations of reading and math tutoring in 12 
states also found that SES’s impact was “modest, 
at best”: while 53 percent of his analyses showed 
directionally positive effects, only 4.4 percent 
found statistically significant positive effects. 
Conversely, 47 percent of the evaluations showed 
directionally negative results, and 3.5 percent 
were negative to a statistically significant degree. 

“We’re so impatient in looking at program effects. 

We want achievement effects now, but is that how 
it really happens with these at-risk kids?” 

— Steven M. Ross 



In part, Meyer and Ross said, such uneven 
results may reflect problems not with the 
tutoring programs but with the evaluations 
themselves, which may be able to control only 
imperfectly for variations in the abilities and 
motivation of enrolled students. But mixed 



results also may reflect the nature of the tutoring 
program, Meyer and Ross said. Students signing 
up for tutoring are already struggling, and they 
may not be eager to take on more schoolwork; 
perhaps as a result, attendance is spotty. In 
theory, market forces should ensure that only 
successful tutoring providers stay in business, 
but parents often lack good information about 
which vendors are effective, and even the same 
vendor may vary in effectiveness, depending 
on how well individual tutors teach and how 
rigorously they follow the vendor’s published 
curriculum. More fundamentally, in the NCLB 
era, effectiveness has been defined narrowly, as a 
statistically significant increase in standardized 
test scores; federal rules dictate that a provider 
who fails for two years in a row to raise scores 
must be dropped from the SES program. But, 
Ross asked, is it reasonable to expect 20 to 
40 hours of tutoring over the course of a year 
to have such a noticeable impact? “We’re so 
impatient in looking at program effects,” Ross 
said. “We want achievement effects now, but 
is that how it really happens with these at-risk 
kids?” Demanding results too quickly may 
mean giving up on providers who, given time, 
could have been effective, he said. In school 
reform, agreed Chafin, the Maryland education 
official, “we know lots of ways to do it right. The 
problem is we keep stopping and starting and 
stopping and starting, over and over and over.” 

Noncognitive Factors 

SES’s true impact might become clearer if 
researchers looked for evidence of the less 
tangible changes — in motivation, behavior, 
study skills — that are likely to precede test- 
score increases, Ross said. Traditional tests 
do not measure such noncognitive skills, and 
NCLB, with its emphasis on reading and math 
scores, has encouraged schools to narrow even 





the band of cognitive skills on which they focus. 
But several conference speakers echoed Ross in 
calling for a new effort to assess noncognitive 
skills, a category that includes such qualities 
as perseverance, leadership, teamwork and 
enthusiasm. Although observers may differ 
about which noncognitive skills schools are 
failing to teach, everyone agrees that something 
is being left out, said conference speaker Elena 
Silva, Senior Policy Analyst at the think tank 
Education Sector. That omission matters 
because measures of noncognitive skills are 
sometimes better predictors of success, in both 
school and career, than traditional cognitive 
measures, said Richard D. Roberts, a principal 
research scientist at ETS. And research indicates 
that noncognitive skills may be easier to modify 
than cognitive ones, he said. “There’s more 
to academic — and, more importantly, job — 
success than is currently represented by the 
cognitive test scores that we get," Roberts said. 

Furthermore, research has shown that 
noncognitive skills can be assessed accurately, 
Roberts said. ETS has begun developing 
such assessments — for example, situational 
judgment tests, which ask test-takers to 
decide, say, how best to organize a study 
group composed of members with conflicting 
schedules. Students applying to graduate school 
now can ask their professors or advisors to 
complete the ETS® Personal Potential Index 
(ETS® PPI), which rates applicants on a range 
of personal qualities, Roberts said. And an ETS 
noncognitive assessment designed for younger 
students generates feedback that can help 
parents and teachers find effective ways to help 
— suggesting, for example, that they avoid early- 
morning discussions of important issues with 
students whom the assessment classifies as 
night people. 



Such work proves that noncognitive skills, far 
from being too fuzzy to assess, “are actually 
quite measurable,” Silva said. And while middle- 
class students acquire these skills outside of 
school, low-income students often do not, she 
said, putting them even further behind once they 
move on to college or career. With the growing 
recognition that noncognitive skills play a role 
in determining children’s futures, “there's an 
urgency behind this now that didn’t exist before,” 
Silva said. 

A Moment to Seize 

After decades spent focusing on schools, the time 
may be ripe for broadening the education reform 
agenda to encompass out-of-school learning in 
all its forms, conference speakers said. The new 
presidential administration appears committed 
to rethinking education policy, and it is backing 
up its priorities with significant investments, 
from stimulus money to Race to the Top grants. 
“If we can't do something about educational 
change now, I’m not sure we ever will be able to,” 
ETS President and CEO Kurt M. Landgraf told 
the conference audience. “We have the first real 
opportunity to change the system, because we 
have the motivator, which is financial support.” 

“If we can’t do something about educational change 
now, I’m not sure we ever will be able to. We have the 
first real opportunity to change the system, because 
we have the motivator, which is financial support.” 

— Kurt M. Landgraf 

Along with the money, however, must go a new 
vision of what education is, conference speakers 
suggested — a vision that, in the Gordon study 
group’s terms, recognizes that education is no 
more confined to schools than health care is to 
hospitals. That analogy between education and 
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public health “moves us closer to the notion that 
education is tantamount to a civil right,” said 
conference speaker Fayneese Miller, Dean of 
the University of Vermont’s College of Education 
and Social Services — a view that “presupposes 
a collective, rather than individualistic, approach 
to education.” 

Thus, changing the system means placing 
education reform in a larger context, said 
keynote speaker Blackwell, the Policy Link 
CEO. By mid-century, a majority of American 
citizens will be people of color, she said, and 
the country will not thrive unless it erases race- 
and income-based disparities in employment, 
life expectancy and educational attainment. A 
unified approach to in-school and out-of-school 
learning is one piece of that larger puzzle, she 
said, and education advocates must connect with 
those working on other aspects of the problem, 
such as health care and economic development. 
“The nation has an interest in investing in people 
who are being left behind,” Blackwell said. “The 
nation doesn’t get to lead in the global economy, 
in the global context, unless all of the people 
who make up the nation can come along.” 

“The nation has an interest in investing in people who 
are being left behind. The nation doesn’t get to lead 
in the global economy, in the global context, unless 
all of the people who make up the nation can come 
along.” — Angela Glover Blackwell 

But turning that vision into on-the-ground 
educational reality — marshaling competing 
ideas about what works, sorting through the 
sometimes conflicting or imperfect data, helping 
communities establish effective programs — 
is a complex task, noted Harvard researcher 
Ferguson. “It may be that it’s fundamentally 
messy, unavoidably so, and that what we need is 
the public will to keep working it through,” he 
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Assistant Dean of diversity; Betsy Brand, Executive 
Director of the American Youth Policy Forum; 

Robert Calfee, education professor emeritus at the 
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said. Rather than do nothing while waiting for 
definitive research on what works, he said, “we 
should privilege common sense until we have 
firmer evidence.” 

Indeed, the question of how best to orchestrate 
out-of-school learning opportunities for children 
does require a measure of wisdom, argued 



psychologist Gordon, who, approaching the age 
of 90, can look back on a career that has outlived 
any number of educational fads. “Even if we 
researchers cannot find ways to measure the 
effect of humane and stimulating and supportive 
development of children," Gordon said, “I would 
still be up here saying common sense says that 
we’ve got to do that.” 
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